Hierarchical Temporal Memory (HTM) networks represent a new approach to machine intelligence. In a HTM network, training data comprising temporal sequences of patterns are presented to a network of nodes. The HTM network then builds a model of the statistical structure inherent to the patterns and sequences in the training data, and thereby learns the underlying ‘causes’ of the temporal sequences of patterns and sequences in the training data. The hierarchical structures of the HTM network allow them to build models of very high dimensional input spaces using reasonable amounts of memory and processing capacity.
FIG. 1 is a diagram illustrating a hierarchical nature of the HTM network where the HTM network 10 has three levels L1, L2, L3, with level L1 being the lowest level, level L3 being the highest level, and level L2 placed between levels L1 and L3. Level L1 has nodes 11A, 11B, 11C and 11D; level L2 has nodes 12A and 12B; and level L3 has node 13. In the example of FIG. 1, the nodes 11A, 11B, 11C, 11D, 12A, 12B, and 13 are hierarchically connected in a tree-like structure such that each node has several children nodes (that is, nodes connected at a lower level) and one parent node (that is, node connected at a higher level). Each node 11A, 11B, 11C, 11D, 12A, 12B, and 13 may have or be associated with a capacity to store and process information. For example, each node 11A, 11B, 11C, 11D, 12A, 12B, and 13 may store sensed input data (for example, sequences of patterns) associated with particular causes. Further, each node 11A, 11B, 11C, 11D, 12A, 12B, and 13 may be arranged to (i) propagate information “forward” (that is, “up” an HTM hierarchy) to any connected parent node and/or (ii) propagate information “back” (that is, “down an HTM hierarchy) to any connected children nodes.
The nodes are associated or coupled to each other by links implemented as hardware or software. A link represents logical or physical relationships between an output of a node and an input of another node. Outputs from a node in the form of variables are communicated between the nodes via the links. Inputs to the HTM 10 from, for example, a sensory system, are supplied to the level L1 nodes 11A-D. A sensory system through which sensed input data is supplied to level L1 nodes 11A-D may relate to various senses (for example, touch, sight, sound).
The HTM training process is a form of unsupervised machine learning. However, during the training process, indexes attached to the input patterns may be presented to the HTM as well. These indexes allow the HTM to associate particular categories with the underlying generative causes that are learned. Once an HTM network has built a model of a particular input space, it can be switched into an ‘inference’ stage. In this stage, novel input patterns are presented to the HTM, and the HTM will generate a ‘belief vector’ that provides a quantitative measure of the degree of belief or likelihood that the input pattern was generated by the underlying cause associated with each of the indexed categories to which the HTM was exposed during the learning stage.
For example, an HTM might have been exposed to images of different animals, and simultaneously provided with category labels such as ‘dog’, ‘cat’, and ‘bird’ that identifies objects in the images during this training stage. In the inference stage, the network may be presented with a novel image of an animal, and the HTM may generate a vector of belief values. Each element in this vector represents the relative belief or likelihood that the novel input pattern is an image of a ‘dog’, ‘cat’, ‘bird’, etc.
The range of pattern recognition applications for which an HTM could be used is very wide. Example applications could include the categorization of email messages as unsolicited bulk email (‘spam’) or legitimate email (‘non-spam’), digital pictures as pornographic or non-pornographic, loan applicants as good or bad credit risks, network traffic as malicious or benign, etc.